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1„ ABSTRACT 

Modem telerobot control technology requires the integration of symbolic and non* 
symbolic programing techniques, different Models of parallel computations, and various 
programing paradt^as. This — paper rtesrrlhmyjfte Multigraph Architecture, which has b een 
developed for the implementation of Intelligentsia! -time control systems^^The layered 
architecture Includes specfffc computational models. Integrated execution ^environment and 
various high-level tools. A special feature of the architecture 1$ the tight coupling 
between the symbolic and non -symbolic computations. It supports not only a data interface, 
out also the integration of the control structures in a parallel computing environment. 




2. INTRODUCTION 

There is an ever-increasing demand for lag) roving/ the Information processing capabilities of robot con- 
trollers, measurement systems, or process control systems. The ultimate goal is to enhance system autonomy* 
adaptivity and functional performance. Since sqme of these features were previously provided by human 
operators, systems exhibiting these properties are/bften qualified to be "intelligent.” 

Essential extension of capabilities Is always based on the application of new techniques. Current trends 
in intelligent systems include: (l) the use cf parallel /distributed computing architectures, (2) the use of 
pare 1 lei /distributed programing models, an<r (3) the application of various artificial intelligence (Al) 
programing techniques. Since the new techniques typically are not substitutes but extensions of the conven- 
tional techniques. Integration has becom^a key factor in building intelligent systems. Symbolic and numerical 
computations, various programing paradfgms and different parallel co^mting models have to be merged in the 
frame of a possibly unified architecture. 

The critical system component /where most of the implementational problems of system Integration have to 
be solved, is the execution environment. The execution environment provides run-time support for the architec- 
ture, and couples various programming models to each other and to the underlying hardware system. 

This paper describes an experimental architecture - the Multi graph Architecture - and the corresponding 
execution environment developed for building integrated systems. After the summary of the background of this 
research, the design considerations are outlined. Then, the main components of the Multi graph Architecture are 
discussed, which is followed/by the summary of various applications and future plans. 


3. BACKGROUND 


A. Intelligent Systems 

There is a rapidly growing research interest in the application of Al techniques in robot controllers, 
measurement systems and process control systems. General, architectural issues of intelligent systems are 
analyzed in [1]. The generic architecture of intelligent systems ; s characterized by the introduction of the 
"know I edge -level ," which includes "knowledge-intensive* system ccxqponents providing high-level perception, 
modelling and planning functionalities. 


The structure and operation of the knowledge-level system components are typically model-driven. New 
possibilities offered by know ledge -based, model -driven automation in telerobotics are described in [2]. Ar- 
chitectural issues of model -driven instrumentation are discussed in [3] and the application of new techniques 
in the Know ledge -based Experiment Builder of a magnetic resonance imaging system is discussed in [4], The 
prospective of model-dri ven, know ledge -based systems in controllers is outlined in [5J. 

A common view regarding the structure of intelligent systems operating in real-time environment is that 
they mjst have layered architecture where "high-level," knowledge-based systea components synthesize, monitor 
and control the operation of the "low-level" sensory and processing activities. 
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B. Graph models of computations 

An important class of parallel computational Models are the graph Models. The coapvtatleoal graphs (or 
control graphs) are directed graphs, where nodes represent units of computations and arcs rep r esent dependency 
relationships. The general properties of graph Models are analyzed in [6]. and their classification 1$ given 
In [7]. In dataflow Models, arcs arise frow data depen d e n ces, and data are passed along the arcs In execution 
tine. In control-flow Models, not the data, but pointers to the data are carried; therefore, this Model re- 
quires the availability of shared memory. The computational units can be scheduled data-driven or deoMd- 
drlven. Data-driven scheduling Means that a unit Is executable If the necessary Input data are available. In 
demand-driven scheduling, only those nodes which are necessary to provide the requested data are activated. 

Graph Models have essential advantages in the context of intelligent real-tine systems. 

- Graph Models can uniformly describe parallel computations for different 
Multiprocessor architectures, such as distributed and shared Memory systems. 

- The granularity of the nodel can be "tuned” by selecting the size of the 
computational units. 

- The "Imperative" parts of the computation (l.e. the code of the computational 
units) are naturally separated from its logic structure represented by the 
control graph. This separation makes it possible to dynamically modify the 
computational structure. 

- The control graph can be easily represented in declarative form. The declarative 
representation is the key for using symbolic processing techniques to synthesize 
various computational structures, such as real-time signal processfng systems [4]. 

4. DESIGN CONSIDERATIONS 

The Hultigraph Architecture (HA) provides software framework for building intelligent systems in real- 
time. parallel computing environment. The main layers of the architecture are: the (1) Physical layer, (2) 
System layer. (3) Nodule layer and (4) Knowledge base layer. The basic properties of the Individual layers are 
summarized below. 


A. Physical layer 

Co^HJtationa? heterogeneity, various physical constraints {such as distance between computing nodes), and 
the typically high computation load require the support of different milt 1 pi e-processor configurations: 
tightly -coup led architectures with shared memory, loosely -coup led computer networks, and their combination. 
Special hardware components such as array processors or l/o devices might also belong to the hardware con- 
f igu rat ions. 


The primary function of the system layer is to provide access Mechanisms to the hardware resources. I« 
the current implementations of MA, the system layers are off-the-shelf operating systems, vhich facilitate 
services such as standard i/o, task management, intertask (interprocessor) communication and synchronization 
and real-time clock. An important requirement for the higher-levels Is flexibility to ensure the portability 
to different operating systems. 


C. Wodule layer 

One of the most critical requirements for HA is to support the synthesis and dynamic modification of 
various low-level computational structures (si9***l processing systems, control systems, etc.) in parallel 
computing environment. The key idea in the solution is the introduction of the module layer, which serves as 
an interface between the knowledge base layer and the system layer. The module layer has a special graph- 
oriented computational model, the Hu Iti graph Computational Nodel (NCN), which provides the following pos- 
sibilities: 


- high-level (possibly very high-level) declarative languages can be defined on 
the knowledge -based layer to represent various computational structures such as 
procedural networks, constraint networks, reasoning networks etc.; 

- these declarative forms can be interpreted and mapped Into a computation graph 
on the module layer; 

- the run-time support of HCH can schedule the elementary computations and “pass* them 
to the system layer for execution, taking advantage of the available parallelism of 
the co*>utational structures; 

- appropriate interpretation techniques can ensure the dynamic modification of the 
computation graph. 
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The MM "Motel* layer* suggests the trie* that this layer Is a "Module library* consisting of typically 
saall prograa Modules written in C, Fortran. LISP etc. These nodules are structured to for* a conplete program 
by the definition of the amputation graph. 

D. Knowledge base layer 

High-level, symbolic amputations are laplenented on the knowledge base layer. Althou*! the actual struc- 
ture of the knowledge-based system components are strongly application dependent, the parallel computing 
environment and the features of the underlying module layer sake the elaboration of a generic programming 
model desirable. The main purposes of the M*-level programming model are: 

- to support the structurization of the knowledge-based operations Into concurrent 
activities. 

- to facilitate a standardized, high-level communication system among the activities, and 

- to provide Interface to the nodule layer and HOI. 


A strict requirement Is that these services have to be implemented as extensions to one of the standard 
LISP systems in order to preserve the compatibility with different AI toolsets. 


5. OVERVIEW OF T* MULT I GRAPH AKH1TECTWUE 

The basic costing models used on the different layers of HA and their relationships are represented In 
Figure l. 



FIGURE 1. Layers of the Multi graph Architecture 
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A. Autjggm ous Co— uni eating Objects (ACO ) 

The basic system structurization principle on the knowledge base layer is provided by the concept of 
Mow— C—ri eating Objects* ACO is a straightforward extension of the "object" concept of object- 
oriented languages such as Flavor [8] in the following sense: 

- ACO's are fully autono— is system that can run virtually or physically parallel, 

- ACO** can be dynamically allocated and can conpete for the sane resources, 

- they co— inicate with each other by neans of a fully asynchronous co— unication 
protocol. 


The naln purpose of ACO's is to provide a standardized "object shell" around a variety of heterogeneous 
knowledge-based system components. The co— ml cat Ion "methods” are standard elements of the object shell, and 
hide the details of an actual Implementation from the application program ers. 

Various object types have been developed for supporting specific applications. These objects typically 
Include a "knowledge base," wffich is represented by a special representation language. Some of these object 
types, such as Procedural network Object (PNO) and Rule Hetwort Object (RNO) are described In [9]. 

B. ttiltl graph Computational Model (HOI) 

The different object -types are facilitated with an appropriate Interpreter or Incremental compiler, which 
maps the actual knowledge base Into a computation graph on the module layer [9]. While ACO's serve as a sym- 
bolic representation and Interface to possibly complex functional components of the system (e.g.» a signal 
processing system, rule-based system, associative database system, etc.), the computation graph on the module 
layer constitutes their actual execution environment. This relationship between the ACO's and their execution 
environment has the following advantages: 

- Execution of the operations represented by ACO's occurs In a parallel execution 
environment offered by NCH. 

- The Interpreter (or incremental compiler) "methods" of ACO's, which build the 
amputation graph, can dynamically modify the graph, as a response to an external 
message, or to a feedback from the execution environment (a mechanism for 
implementing "self- modifying" signal processing systems is described In [4]). 

- The system fully Integrates two different parallel computing models. ACO’s form 
the "macro-structure" of the system, and they communicate by using the services of 
loosely coupled distributed systems (typically message passing). The computation 
graphs provide the "micro-structure" of the system components. MCH efficiently 
supports medium-level (subroutine size) computational granularity, and can take 
advantage of tightly -coup led multiprocessor architectures with shared memory. 


C. System tasks 

The computational model on the system layer is provided by the actual operating system. The key concept 
is the "system task," which represents a "slice" from the processing capacity, and can access to various 
resources. The elementary computation units that are scheduled by the run-time support of MCH are executed by 
the system tasks. 

6. HULTIGRATH COMPUTATIONAL NOOEL 

MCM can be characterized as a control-flow model. The control structures of computations are represented 
by bipartite graphs that are built of actomodes, datanodes and connection specifications (see Figure 2). 

The actomodes are associated with the elementary computational units, called scripts, which can be 
written either in LISP or in any other language, such as C, Fortran, Pascal, etc. The scripts do not know 
about their position in the control graph: they communicate with other graph components through the 
input/output ports of the actomodes. The actomodes are associated with a local datastructure, called 
context, which can be accessed by the script* If the code of the script is reentrant, it can be attached to 
several actomodes. In different computation problems the scripts may be quite different: a script may be an 
interrupt -driven l/o handler, a transformation of the input data arriving to the actornode, or an Interpreter 
module, which Interprets the symbolic form stored in the context of the actornode. 

Datanodes store and pass the data generated by actomodes. They can be either streams with wltiple 
output ports, or scalars. The streams maintain the partial sequence order of the data generated during the 
computations, irfiich preserves the overall consistency. 

The control graph can be operated in data-driven or in demand-driven mode, or in a combination of the two 
modes. In data-driven mode, the data sent to a datanode propagate a "control token" to the connected actor- 
nodes. The actomodes will fire according to the specified control discipline: in ifall mode, at least one 
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FIGURE 2. Conponents of the Multi graph Computation Model 


control token wst be sent to all of the Inputs of the actomode: In Ifany mode, every received control token 
will cause firing. In demand-driven mode, the request for data sent to a datanode will generate a control 
token If the datanode Is “eppty." This control token will fire all of the actomodes that are potentially able 
to provide the requested data. The demand propagates backward along the control graph until data Is generated. 
From this point a forward propagation starts, which finally provides the requested data. A more detailed 
description of the computational model can be found In [XO, XI]. 

The mn-tlme support for the HCM Is provided by the Multi graph Kernel (*C). The structure of the MC Is 
shown In Figure 3. The control graph Is represented In the descriptors, tdilch are manipulated by various 
kernel functions. The Control Interface functions are used for dynamically building and modifying the control 
graph. These functions are Imbedded In a LISP system, where the various graph-builder Interpreters and In- 
cremental compilers are Implemented. The Nodule Interface Includes the data/demand propagation kernel calls 
for the scripts. An Important feature of the system Is that actomodes with scripts written In different 
languages can be mixed in the same control graph. (The necessary transfer routines are invisible to the user.) 
This feature Is used for creating tight coupling between symbolic and non -symbol 1c computations. Ti^it cou- 
pling means that not only data structures can be passed between the two kinds of computations, but there may 
be a fully integrated control structure. 


CreateActor( ): CreateOata( ): 



FIGURE 3. Structure of the Multigraph Kernel 
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The most complex part of MC Is the System Interface. The System Interface schedules the elementary com- 
potations that are defined during the data /demand propagation. The conputatlonal units (the scripts of the 
fired actomodes) are passed to the available systen tasks for execution. The mlr— rut n ecfc a n lsa of the 
Systeu Interface ensures that subsets of the control graph can be <frna*tcally associated with one or more 
systen tasks that Include the necessary resources for executing the scripts. This mechanism provides a very 
straightforward way for dynamic resource aanagenent In nultlprocessor configurations. 

Two I^KHtant Inpleuentatlon Issues are the granularity and the memory aodel. The lower Unit for the 
reasonable conputatlonal granularity Is basically determined by the overhead of I9C. Since MC currently Is 
1^>1enented In software, the overhead Is Introduced by the control token propagation functions. On the 68000 
processor-based IBM 9000 system (clock frequency Is Miz), the overhead Is about 800 microseconds: on the VAX 
78S Implementation Is less then 200 microseconds. Due to the construction of the NX. the overhead Is basically 
Independent from the size of the control graph. 

Since PCM Is a control -graph model where the pointers to data structures rather than the data are passed 
along the graph, the model requires the presence of shared memory for those tasks (and processors) that are 
assigned to the same subgraph as execution resource (see Figure 4). In order to provide flexibility toward 
architectures which do not support shared memory access for the processors (hypercube architectures or dis- 
tributed computer configurations), a simple mechanism Is Implemented to link control graphs that are allocated 
In the local memory of the separate nodes. The scripts of receiver and tr a ns mitt er actomodes provide a logi- 
cal link between the separated subgraphs and Implement the data transfer fay using the actual services of the 
underlying system layer (e.g. , the message passing services of DECNET in the VMS/DECMET implementation). At 
these links, the control-graph model Is "transformed - to dataflow model, since the actual dat as t ruct u res - and 
not just pointers - are passed to the "remote" nodes. 

This method makes It possible to generate large processing networks from their symbolic representation In 
distributed computing environment [12]. 


NODE -1 WE -2 



FIGURE 4. Memory Model of MCM 


7. STRUCTURE OF THE EXECUTION ENVIRONMENT 

The simplified structure of the execution environment supporting MA can be seen In Figure 5. MX is imple- 
mented as an additional layer to a standard operating system. Depending on the computer architecture and on 
the features of the particular operating system, I* may exist in one or more copies. The System Interface of 
MX is a well structured, modular program which makes porting the kernel relatively easy, even to devastatingly 
different operating systems and real-time supervisors. 

The Module Interface functions of MX can be Invoked from LISP as well as from other languages. This 
ensures that scripts can be written In different languages, and existing module libraries can be easily inter- 
faced to liC. The Control Interface of MX is imbedded in LISP since currently we use LISP as implementation 
language of the knowledge base layer. 

Various high-level software components such as the generic object shell for ACO’s and the standard 
methods of different ACO types are implemented in LISP. 

For distributed computer configurations, the LISP system (in the first Implementation FRANZ LISP, 
recently changed to Zamon Lisp) has been expanded with the Comnunicating LISP System facility, which provides 
task management and asynchronous message passing primitives [13]. 
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FIGURE 5. Structure of the Execution Environment 


8. EXPERIENCES 

MA has been i«*>lemented on very different computer configurations and has been used for various applica- 
tions. 

An Intelligent Test Integration System (HIS) has been implemented in the Space Station laboratory of the 
Boeing Aerospace Company In Huntsville* AL [12]. The purpose of ITIS is to support the automatic generation of 
test systems in real-time* distributed co*>uting environment. ITIS is implemented as a knowledge base layer 
above the conventional test system components, and can build co^>lex test configurations from the symbolic 
specification of test scenarios. The computing environment is a VAX network with the VMS /BECHET operating 
system expanded with special hardware units. 

A different application of the layered HA architecture is the Knowledge-based Experiment Builder (KEB). 
which was developed for the M.I.T/IBM experimental HR I (Magnetic Resonance Imaging) system [4]. The core of 
the KEB is a high-level representation language for signal processing schemes and a smart interpreter* trfiich 
can generate the appropriate version of the real-time signal processing system* and which is able to recon- 
figurate it for specific events. The computing environment of this system is the IBM 9000 computer with the 
CSOS real-time operating system. 

Various computational structures have been developed and are being investigated for tne HCM, such as a 
hierarchical planner [14], knowledge-based similation builder [15], pattern-driven inference system [16], etc. 

9. CONCLUSIONS AND FUTURE PLANS 

The integration of symbolic and conventional programing techniques, parallel computing models of dif- 
ferent granularity, and various programming paradigms are essential conditions for the successful implementa- 
tion of intelligent real-time systems. The Multigraph Architecture has proven to be a good approach to solve 
the problems of integration. It provides a generic framework, programming models for stnjcturizing software 
components and various tools for the actual implementation. 

We have practical experiences with implementing systems in single -process or (IBM-AT/MS-DOS), single- 
processor multitasking (VAX/VMS and IBM 9000/CS0S) and distributed (VAX network YMS/0ECJ4ET) computing environ- 
ments. As a next step, we intend to implement the execution environment for tightly-coupled mil ti processor 
configuration. 

The system offers a convenient method to describe and implement "sel f -modi fying“ signal processing sys- 
tems. Further research is needed to utilize this capability in tne design of structurally adaptive measurement 
control systems. 
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